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rehospitalization. These three criteria were; (1) satisfaction with 
self, (2) satisfaction with interpersonal relations, and (3) 
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eighteen item scale was developed to measure the present state, and 
the magnitude and direction of change of the three criterion areas. 
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The assessment of the outcome of psychotherapy has been of major concern to 
social scientists, especially in the past twenty years. Frequently researchers 
have raised important questions about the effectiveness of specific psycho- 
therapies and psychotherapy in general. Eysenck (1952-1965) questioned whether 
anyone had adequately demonstrated the effectiveness of any specific type of 
psychotherapy or psychotherapy in general. Many authors, in reviewing this body 
of research, have agreed that in the following years since Eysenck's classical 
paper, little adequate research was conducted in the tryriad of studies published 
in the area of evaluation research (Zax and Klein, 1960; Strupp, 1964; Shlien, 

1966; Paul, 1967). 

THE CRITERION ISSUES 

The issue of criteria selection in therapy evaluation has long been addressed 
and readdressed (Bartlett, 1959; Kaczkowski and Rothney, 1956; Strupp and Luborsky, 
1962; Strupp, 1963, Frank, 1963; Zax and Klein, 1960; Farnsworth, 1966; Ellsworth, 
1968). After review of the many studies dealing with problems of criteria selec- 
tion, some attempt at a summary of the issues involved was considered in a number 
of articles (Frank, 1963; Farnsworth, 1966; Paul, 1967). Farnsworth (1966) and 
Frank (1963) both indicated that criteria for evaluation should now be relatively 
clear and it is time for consolidation and implementation of those criteria 
using modern psychometric techniques. This paper was just such an attempt at 
scale development. This presentation limited its scope to the development of 
multiple rater assessment measures designed to evaluate treatment effectiveness. 
Much previous research has dealt with assessment by trained professional inform- 
ants (Lorr, 1954; Zax and Klein, 1960; Farnsworth, 1966). The problems of assess- 
ment of the therapeutic experience by the personnel providing the service is 
O 



obvious in the inherent bias introduced by such procedures (Zax and Klein, 1960, 
p. 439). With these inherent biases in mind, the present author attempted to 
develop a research tool that can be applied to any number of therapeutic 
approaches and settings. 

MULTIPLE CRITERIA AND MULTIPLE RATERS 

The use of single criteria in previous research has not proved a successful 
tactic whether it be derived from professional or layman. Empirically, the 
singular use of self-satisfaction, self-concept, self-knowledge, test score 
changes and number of other such ratings have not been adequate (Block and 
Thomas, 1955; Forsythe and Fairwaather , 1961; Taylor, 1955; Froehlick, 1957; 
Tyler, 1961; Zax and Klein, 1960). Strupp and Luborsky (1962) summarized the 
Issues against the use of a single rater c? any kind, be it professional or non- 
'professional , trained or untrained. Warnings against the use of the patient 
himself as the sole criterion measure have been expressed by a number of authors 
and researchers (Bardie, 1954; Berg, 1952; Farnsworth, 1966; Tyler, 1961; Strupp, 
1963; Zax and Klein, 1960). In a core article, Goodstein and Grigg (19 rr 
emphasized the need for multiple criteria measures and multiple raters. They 
postulated that in evaluation research there is need fo" consideration of 
personal and social adjustment by the client and significant others, as well as 
.actual performance records, and client satisfaction with the therapeutic process. 
It might further have been said that criteria variables may be rated by at least 
three different types of raters: (1) hospital or professional personnel; (2) 
the client; and (3) a relative or friend (some significant other). This emphasis 
on multiple and non-professional raters has been borne out empirically by some 
recent studies (Ellsworth et al., 1968; Katz and Lyerly, 1963; Michaux . 



et al . , 1969; Vestre and Zimmerman, 1969), It has, therefore, been determined 

by previous research that the task appeared to be the development of a multiple 

criterion measure that can be rated by at least three different informants. The 

question now remains as to which of the many multiple criteria relate best to 

other individual outcome measures and which have been empirically validated in 

0 number of adequate research studies. An additional limitation on the type of 

criteria exists in the need for criteria that can be adopted to any theoretical 

..approach and therapeutic setting. 

\ 

\ 

self and multiple assessments 

The first evaluation measures based on the client’s assessment of the thera- 
peutic situation ware developed for use in client-centered therapy. These phenom- 
Onologi c^ measures were used intra-therapeutically and post-therapeuti cally. 

w 

Seeman (1954) constructed a measure which was used in a number of client-centered 
Research studies (Rogers and Dymond, 1954). It consisted of ten nine-point scales 
Which required the counselor to note different aspects of the c^ien*" ^ exr 
While **hi s instrument was not a self-assessment measure, its findings prove 
interesting when related to self-assessment measures. Using Seaman's scale as both 
a process and a oefore-after '.ease re of client-centered therapy, it was found that 
success of outcome was relaiod to the extent to which the client used the thera- 
peutic experience for personal exploration, liked and accepted his therapist., 
moved in the direction of personal and social adjustment, and was satisfied with 
the outcome of therapy. These last three findings, personal and social adjustment 
and satisfaction with the cut;.>me cf therany form the core criteria in a n nnber 
Of Outcome stu ies. 

The earliest, adequate phenomenological seif-assessment measure to be used in 
a number of studies (Snyder, 1953) was developed by Tucker (1953). It was an 
pnVWellent attempt at the use of mu tiple criteria in therapy evaluation. Tucker 
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called his measure tha"client post-therapy scale" on -which the client noted his 
feelings in several areas: (a) possibility of having problems in the future; 

(b) studies of problems which prompted treatment; (c) relationship to others; 

(d) relationship to immediate family; and (e) sexual adjustment. 

A more recent phenomenological approach to self-assessment measures was con- 
tributed by Katz and Lyerly (1963). The Katz Adjustment Scale (KAS) was designed 
to be used by both the client himself and significant others in rating the patient's 
symptoms and his personal and social behavior. There are 127 items which cover 
typical aspects of psychopathology and are to be judged on a 4-point scale accord- 
ing to frequency of occurrence. The KAS was divided into a patient form and a 
community informant form, each of which was identical except for a longer com- 
munity informant synnten scale. The community informant sympton discriminant 
(SD) scale had 123 items, while the patient form only had 55. Of the five 
seal £*3 developed ror the patients and the six developed for the community 
informant only satisfaction with socially accepted activities (SSA) and per- 
formance of soci all y expected activities (SEA) for both patient and informant 
predicted rehospitalization. In addition the "social" measures again best pre- 
dicted course of adjustment. Another scale labeled " satisfaction with free time 
activities" (SFA) which actually described satisfaction v/ith individual and social 
activities and hobbies, also predicted course of adjustment. 

Zax and Klein (1960) emphasized the evaluation of the patient's behavior 
outside the therapy situation as the important approach to the criterion issue. 
Indeed, more recent articles indicate that there is little congruence between 
community and hospital adjustment (Ellsworth et^ al_. , 1968; Wood et al . , 

1962; Buss ert al_. , 1962; Schooler et_ al_. , 1967; Sinnett ert al_. , 1965). 

Therefore, the inportance of ratings done in the conmunity and by the 
ommunity members is highly indicated. Rogers and Dymond hypothesized the 



independence of vantage points from which therapeutic change was observed (Rogers 
and Dymond, 1954). Cartwright and his associates, however, found that some agree- 
ment between independent observers does occur even though patient and therapist 
ratings consistently appear most disparate (Cartwright and Roth, 1957; Cartwright 
and Kirtner and Fiske, 1963). 

The domains of criteria for study were to be ones that exist outside the 
therapeutic situation itself. Criteria domains most consistently studied in the 
past and best related to traditional outcome variables such as length of stay, 
rehospitalization, therapist and relative ratings, have generally been in three 
basic areas. While these traditional outcome variables have been given much 
attention, many problems are inherent in their usage (Ellsworth et al_. , 1968). 

The three criteria domains that have most consistently been linked with 
successful erapy have been patient satisfaction or contentment with himself , 
his interpersonal life and satisfaction with the outcome of his therapeutic 
experience (Crown, 1968; Ellsworth et al_. , 1968; Katz and Lyerly, 1963; Lewinsohn 
and Nichols, 1964, 1967; Meltzoff and Blementhal , 1966; Nichols and Beck, 1960; 
Tucker, 1953). Indeed, in an excellent review of psychotherapy research Paul 
(1967) stated that "the real question of outcome on logical and ethical grounds 
is whether or not the clients have received help with the distressing behaviors 
which brought them to treatment in the first place" (p.112). Likewise, in 
another well done review of the "healing arts," Frank (1963) indicated that 
the appropriate criteria for psychotherapy research should be reduction of 
distress and improvement in functioning. Successful functioning, Frank des ~ 
cribes as "not merely a matter of behavior but of attitudes, so evaluation of 
this criteria must take into account not only the patients behavior but its 
meaning for himself and those who are important to him" ( p . 2 30 ) . In a study 



which simply asked the patient what his treatment was centered about. Board (1959) 
found that almost all patients saw improvemnet in self-understanding and various 
interpersonal relationships as the core of his distressing problems. 

MEASUREMENT OF CHANGE 

The criteria domains were therefore established and the next concern was the 
assessment of change within the particular domains. The literature on the assess- 
\ ment of change in psychotherapy presents a complex of issues that must be attended 
to before "change" can be established. The GAP Report (1966) presented an excel- 
lent review chapter on the assessment of change. They presented five basic dimen- 
sions of change: (1) occurence (simply yes or no); direction (usually better or 

worse) s\(3) magnitude (adds "how much" to direction); (4) duration; and (5) 
reversibility 

Likewise, in other articles on the assessment of change in psychotherapy, 
Meltzoff and Blumenthal (1966) and Meltzoff and Kornreich (1970) concluded that 
change has (a) a content statement; (b) direction; (c) magnitude; and (d) rate 
of occurrence, among other lesser attributes. Because there was considerable 
overlap between some of these dimensions, three of the most independent, salient 
and most agreed upon areas were chosen: (1) occurence; (2) magnitude; and (3) 
direction. Having, therefore, found some important dimensions necessary for 
the establishment of change, only the application of these dimensions of change to 
the three previously established criteria domains was needed. 

DEVELOPMENT OF ITEMS 

The task was to develop items which would assess the three criteria domains 
and dimensions of change as shown in Figure I. By perusal '.of past scaling attempts 
and consultation of staff members at the Fort Logan Mental Health Center, a 

^number of items in each of the nine areas were developed. The items were then 

— 6 “ ^ 



subjected to some prior testing on adult psychiatric patients at the aforementioned 
institution and patients were interviewed about their clarity, appropriateness 
and meaningfulness. Word changes clarifying item meanings were made resulting in 
the eighteen items presented in Table I. 



Satisfaction with therapy and outcome 
Satisfaction with self 

Satisfaction with interpersonal relations 



CHANGE PRESENT 

MAGNITUDE DIRECTION STATE 
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Figure 1. The Domains and Dimensions of the Treatment Effectiveness Scale 
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(This scale not to be reproduced without permission of the author) 



Table I 



1. SA TISFACTION WITH THERAPY (DIRECTION AND DEGREE OF CHANGE) 

To what extent have your complaints or problems that brought you to therapy 
changed as a result of treatment? 

Very Greatly Considerably Somewhat Not at all Gotten 

Improved Improved Improved Improved Worse 

Everything onsidered how satisfied are you with the changes in yourself that 
have occurred as a result of your experience at Fort Logan? 

Not at All Not Very Somewhat Rather Well Very Well 

Satisfied Satisfied Satisfied Satisfied Satisfied 

In general, how satisfied are you with your experience at Fort Logan? 

Not at All Not Very Somewhat Rather Well Very Well 

Satisfied Satisfied Satisfied Satisfied Satisfied 

In general, did you find your experience at Fort Logan worthwhile? 

Extremely Very Somewhat Not at all A Bad 

Worthwhile Worthwhile Worthwhile Worthwhile Experience 

How sure are you that you can nov/ handle your problems without Fort Logan's 
help? 

Very Somewhat Pretty Not Very Not at all 

Sure Sure Sure Sure Sure 
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II. SATISFACTION WITH SELF-MAGNITUDE 



How much change in you as a person has occurred since you started treatment at 
Fort Logan? 

Not Very little Somewhat A Good Deal Very Much 

Changed Changed Changed Changed Changed 

How much do you feel you have changed as a result of Fort Logan? 

Great Fair 

Deal Amount Somewhat Very Little Not at All 



SATISFACTION WITH SELF-DIRECTION 

What change has there been in your feelings about yourself? 

More No Somewhat More Good Deal More 

Discontented Change Contented Contented 



Much More 
Contented 



What change has there been in your feelings toward your life? 

More No Somewhat More Good Deal More 

Discontented Change Contented Contented 



Much More 
Contented 



SATISFACTION WITH SELF— PRESENT STATE 

How Satisfied are you with your life right now? 

‘Very Somewhat Neither Satisfied Somewhat Very 

Satisfied Satisfied Nor Dissatisfied Dissatisfied Dissatisfied 



How do you feel about yourself? 
Very Somewhat 

Contented Contented 



Neither Contented Somewhat Very 

Nor Discontented Discontented Discontented 
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SATISFACTION WITH SOCIAL - MAGNITUDE 



How much change has occurred with the way you get along with others since your 
treatment at Fort Logan? 

Mot Very Little Somewhat Good Deal Extremely 

Changed Changed Changed Changed Changed 

How much change in your social relations has occurred as a result of your 
treatment at Fort Logan? 

Great Deal Fair Amount Somewhat Very Little Not at all 



DIRECTION - SATISFACTION VJITH SOCIAL 

I am more well thought of and respected by others since my stay at Fort Logan. 
Strongly .Neither Agree Strongly 

Agree Agree Nor Disagree Disagree Disagree 



Are ymj generally more content with the way you act with other people since 
the FoVt Logan experience? 

More Somewhat More More Much More 

Discontented No Change Contented Contented Contented 



SATISFACTION WITH SOCIAL - PRESENT STATE 

How satisfied are you with the way you presently get along with others? 

Not at all Not Very Somewhat Pretty Much Very Well 

Satisfied Satisfied Satisfied Satisfied Satisfied 

How content are you with the way you presently make and maintain friendships 
with others? 

Very Pretty Much Somewhat Not Very Not at all 

Content Content Content Content Content 

What changes have there been in your feelings toward your interactions with 
others? 

More Somewhat More More Much More 

Discontented No Change Contented Contented Contented 



PRELIMINARY ANALYSIS 



The eighteen items were furnished with a cover letter fully explaining its 
use and privacy of communication and mailed out to former patients and their 
community informants. In addition to the eighteen items presented, four items 
taken from a previously developed questionnaire assessing feelings about staff 
concern and rapport and staff competence “eluded in the questionnaire. 

The wenty-two items were randomized and .w’lec out in a questionnaire. 

Analyses were deducted on the four slj* lies (including perceived staff 
concern and competence). The reliabilities, homogeneity ratios and interscale 
statistics for 86 subjects (include both patient and conmunity informants) are 
presented in Table 2. 

Table 2 



Scaling Statistics for TES Subscales 

1 2 3 4 5 



1 . 


Satisfaction with therapy outcome (STO) 


(.88) 


- 






2. 


Satisfaction with self (SWS) 


.86 


(.92) 






3. 


Satisfaction with interpersonal relations (SIR) 


.78 


.88 


(.91) 




4. 


Staff concern and competence (SCC) 


.•67 


.50 


.39 (.85) 




5. 


Total treatment effectiveness scale (TES) 
(Not including SCC) 


.91 


.97 


.95 .53 (.96) 




Homogeneity Ratio 


.60 


.66 


.61 .58 


.59 




Number of Items 


5 


6 


7 4 


18 



As can be seen in Table 2, all of the TES subscales and the total TES scale 
had high reliabilities and very high homogenei ty. ratios indicating exceptional 
internal consistencies. Indeed, the reliability for the total TES was .96. The 
homogeneity ratio was .59 which is somewhat higher than the optimal level of ,33 
which indicated some redundancy and overlap which was expected considering the use 
of multiple dimensions of change having the same content refe. jut .n'thin each sub- 
scale. There were only minor differences in items assessing me. ani t :de , rection 

and present state of each criterion domain. The high correlatic ~ ’r twee SCC and 
the other TES scales is in agreement with previous research which .dicated a high 
relationship between patient feelings of staff concern and compete: a an. satis- 

faction with the outcomes of therapy, self-appraisal and interpers ^.1 T = e 
(Cartwright, 1957; Kamin and Caughl in ,1963; Luborsky et_ al_. , 1971) All f the 

preceding and following analysis was done with both patients and community infor- 
mants. No differences were observed in the separate patient and community infor- 
mant analyses, therefore only the combined analyses are presented. 

In addition to the more or less a p riori scaling analysis, these 22 items 
were subjected to standard empirical cluster analysis (BC Try system). The results 
of this analysis are presented in Table 3. Fourteen of the eighteen items in the 
total TES clustered empirical with three unincluded items forming a third cluster 
which is highly related to the first cluster (r=.78). The SCC scale empirically 
clustered as the second cluster seen in Table 2 with the addition of one STO 
item, which supports both scales internal consistencies and the rationale for 
considering SCC as separate from the TES itself. This separate clustering of 
the SCC from the TES also indicates a less likelihood of such empirical clustering 
due only to a "method factor" (Cartwright, Kirtner and Fiske, 1963) or a"vantage 
point" factor (Rogers and Dymond, 1954). 



O 
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TABLE 3 
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EMPIRICAL CLUSTER ANALYSIS FOR 


TES 


ITEMS 






ITEM TYPE 


VARIABLE NO. 1 


FACTOR COEFFICIENT 


AVERAGE WITH (D) 


CLUSTER 


SIR-DIR 


19 (D) 2 




.88 


.74 


NO. 1 












SWS-MAG 


2 (D) 




.87 


J’' 




SWS-MAG 


10 CD) 




- .87 


.73 




SWS-DIR 


12 (D) 




.86 


c 




STO 


9 CD) 




- .84 


.71 




SWS-DIR 


22 CD) 




.83 


.70 




SIR-DIR 


6 (D) 




.81 


.68 




STO 


18 




.81 


.68 




SWS-PS 


11 (D) 




- .75 


.63 




SIR-MAG 


4 




.74 


.63 




SWS-PS 


3 




.74 


.63 


) 


SIR-DIR 


15 




- .72 


.61 


STO 


21 




- .70 


.59 




STO 


20 




.60 


.50 






RELIABILITY OF CLUSTER = 


.95 






CLUSTER 
NO. 2 


see 


16 (D) 




.88 


.66 




see 


17 (D) 




- .83 


.63 




see 


8 




- .71 


.54 




STO 


1(D) . 




.70 . 


.54 




see 


7 (D) 




— .61 


.47 






RELIABILITY OF CLUSTER 2 , 


,88 






CLUSTER 
wn. 3 


SIR-PS 


5 CD) 




.89 


.78 




SIR-PS 


14 (D) 




- .86 


.76 




SIR-MAG 


13 




- .66 


.58 






RELIABILITY OF CLUSTER = . 


85 







^ACTUAL ITEMS ARE LISTED IN APPENDIX .14 
Z INDI CATES DEFINERS OF A CLUSTER 




VALIDATION 



The next step in the process of scale development is the validation of the 
scale. Since community informants are becoming a generally used informant popu- 
lation in therapy research, it was decided to validate the TES by correlating 
patient and informant ratings on the TES. Only 15 complete data sets were 
available at this date in the analysis (both patient forms and their community 
informant forms completed). Pearson r statistics were computed on the total 
TES scale, the three subscales and SCC scale. Table 4 gives the correlations 
between the five above mentioned scales for patients and their respective 



:ommufiity informants. 

\ 



Table 4 

Correlations Between Patients and Community 

Informants on the TES a nd__bU. 



SCC 

SWS 

SIR 

STO 



.60** 

.77*** 

.58* 

.71** 



TCC 






~ * p < .05 

** p <.01 

***p <.001 

While the N in this initial validation procedure is small, nevertheless the 
statistics are still impressive. The two informant populations rate the TES 
scale very similarly indicating high convergence validity for the overall scale. 
The SWS has the highest convergence followed by the STO and SI., subscales, 
indicates that community and patients agree more with regard to satisfaction 
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wit sel u*tan satisfaction i th therapy or interpersonal relations. This 
finding is reasonable since informants can generally better observe the 
patien-'s self-evaluation than his interpersonal relations which may exist 
outside the informant's social environment. 

In essence, the cross-validation procedure indicated good correspondence 
between patient and corrsnuni ty informant. However, a larger sample, as well 
as other cross-validation procedures, are still needed and are the next steps 
to be taken by the present author. 
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